Skip to content

Aggregate flag evaluations on the evaluating thread (FFE optimization) - #13622

Draft
dougqh wants to merge 2 commits into
dougqh/ffe-context-hashfrom
dougqh/ffe-caller-side-aggregation
Draft

dougqh wants to merge 2 commits into
dougqh/ffe-context-hashfrom
dougqh/ffe-caller-side-aggregation

Conversation

@dougqh

@dougqh dougqh commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Stacked on #13609. It also carries the commit of #13612 (heap-aware ConcurrentHashtable bucket sizing), which drops out once that lands.

Flag evaluations are now aggregated on the evaluating thread, into toolbox ConcurrentHashtables. This replaces allocating a FlagEvalEvent per evaluation and queueing it for a single worker thread. It's the only path; there's no switch.

  • Recording: an evaluation that lands in an existing bucket costs a lock-free chain scan and a few atomic updates, and allocates nothing:

    • no composite key: the lookup hashes and compares the key parts directly;
    • the table's entry is the row itself;
    • no event object.

    A new bucket is inserted through a Reservation under the table's write lock, which also enforces the tier caps. The tiers, caps and bucket keys are unchanged (full tier, per-flag cap, degraded tier with the consent fold), so the payloads don't change.

  • Draining in place: buckets live across flushes. A bucket whose count goes from 0 to 1 pushes itself onto a lock-free stack through its own dirtyNext field. The flush takes the stack and reads and resets each bucket's counters. An evaluation racing the flush is counted in this flush or the next one, never lost. Buckets idle for a whole window are swept out, freeing their slots under the caps.

  • The worker only flushes, every flush interval and at shutdown. It waits on the shutdown latch rather than polling a queue. Evaluations recorded after close are counted as closed drops.

Deleted: FlagEvalEvent, FlagEvaluationAggregator, the evaluation EventQueue and its overflow metric, and FlagEvaluationEnqueueContentionBenchmark. EventQueue stays for exposures.

Motivation

Under load, the single worker couldn't keep up with the queue: the unpaced client benchmark dropped 29–67% of flag-evaluation events. The ConcurrentHashtable works as a queue that merges same-key events on arrival, so its size depends on the number of distinct rows, not on the event rate.

A prototype of this design was measured on the FFE roll-up by a benchmark worker (ClientEvaluationBenchmark, JDK 21, one jar, queue path against caller-side, 3 forks each):

  • Equal work (paced, 4 threads, the queue dropping about 1%), where the time per evaluation includes the pacing delay, so only the difference counts:

    case time bytes per evaluation
    flags 0.0% 916 → 810 B
    splits −1.1% 1,642 → 1,512 B
    variations +1.1% (noise) 27,535 → 27,384 B
  • Drops: 0.0% in every unpaced fork, against 29–67% on the queue path. Unpaced timings aren't comparable, because a dropped evaluation skips the hook's work.

  • Exact counts: at 8 threads, the posted evaluation_count matched the evaluations performed: 223,638,646 = 223,638,646 for flags, and splits and variations matched too. The queue path posted 68.8M of 213.1M.

  • 128 MB G1 heap: flags throughput +17%, and GC pause time per million evaluations 24.4 → 12.8 ms.

  • Hot path: the aggregator is 4.8–6.9% of evaluating-thread time, with no contention frames. A flush takes about 100 ms per 10 s window.

Additional Notes

🤖 Generated with Claude Code

dougqh and others added 2 commits November 21, 2026 14:31
createBounded allocated one bucket per entry of its cap, up front and for
good, since the table never resizes. A table capped at 131,072 entries
retained a 512 KiB array even while nearly empty, which on a small heap
is out of proportion and, under G1, a humongous allocation.

Size bounded tables by bucketCountFor(maxCapacity) instead: one bucket
per entry for small tables, but at most 1/1024 of the maximum heap and
below G1's humongous threshold, with a floor of 1,024 buckets. The cap
is unchanged; chains carry any excess. A new overload takes an explicit
bucket count for callers that know their expected population.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each flag evaluation used to build a FlagEvalEvent and offer it to a
bounded 4,096-slot queue, which a single worker thread drained into the
aggregator. Under load the worker could not keep up, and the queue
dropped 30-67% of evaluations.

Record evaluations straight into a ConcurrentFlagAggregator instead. An
evaluation that lands in an existing bucket costs a lock-free chain scan
and a few atomic updates and allocates nothing; a new bucket is inserted
under the toolbox ConcurrentHashtable's write lock, which also enforces
the tier caps. The tiers, caps and bucket keys are unchanged, so the
payloads are the same.

Buckets are drained in place: a bucket whose count goes from zero to one
pushes itself onto a lock-free stack, and the flush reads and resets the
counters of the buckets on it. An evaluation racing the flush is counted
in this flush or the next one, never lost. Buckets idle for a whole
window are swept out, freeing their slots under the caps.

The worker now only flushes, every flush interval and at shutdown.
Evaluations recorded after close are still counted as closed drops, and
close() counts those that land after the final flush. The queue-overflow
drop metric goes away with the queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 21, 2026
@pr-commenter

pr-commenter Bot commented Nov 21, 2026

Copy link
Copy Markdown

LLM Validation

LLM Validation Gate — java-agent-instructions

✅ PASS

  • No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

6 case(s) · quality 81.2 → 81.2 (+0.1) · win-rate 49% [46%–51%]

Details — analysis and per-case results

Analysis

Changed instruction file(s): AGENTS.md.

No safety or blocking-case regressions across 6 case(s). Overall pairwise win-rate 49% [46%–51%], quality +0.1 — see the verdict above for whether that clears the noise band.

Results

  • Pairwise win-rate: 49% [46%–51%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
  • Overall quality: 81.2 → 81.2 (/100, +0.1)
  • Bad signals introduced (advisory): 0
  • Candidate criteria coverage (advisory): 22/27 (81%) — expected_criteria the candidate met; does not affect the gate
  • Blocking-case regressions: 0

Cases

Case Mode Quality Δ Win-rate (90% CI) Safety
java-core-validation-001 warn -1.7 50% [50%–50%] ok
java-unit-test-conventions-001 warn -1.1 50% [50%–50%] ok
java-premain-safety-001 warn +3.1 50% [50%–50%] ok
java-ci-download-guidance-001 warn +1.9 50% [50%–50%] ok
java-documentation-only-validation-001 warn -1.1 42% [28%–55%] ok
java-existing-spock-maintenance-001 warn -0.6 50% [50%–50%] ok

Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 98.68%
• Overall Coverage: 60.77% (-0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a54a61d | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.04 s 13.94 s [+0.1%; +1.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.94 s 12.94 s [-0.7%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.26 s 17.15 s [-0.1%; +1.4%] (no difference)
startup:petclinic:iast:Agent 17.05 s 17.22 s [-1.6%; -0.4%] (maybe better)
startup:petclinic:profiling:Agent 16.83 s 16.95 s [-1.8%; +0.4%] (no difference)
startup:petclinic:sca:Agent 17.22 s 17.05 s [+0.2%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 15.77 s 16.43 s [-8.2%; +0.2%] (no difference)

Commit: a54a61d4 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant